Working with AWS CloudTrail

Overview

I created an AWS CloudTrail trail to audit actions taken in my account. I then investigated to determine who modified the Café website.

This started with an Amazon Elastic Compute Cloud (Amazon EC2) instance named Café Web Server, which runs a web application that hosts the Café website.

In Task 1, I observed that the website looked normal.

In Task 2, soon after creating a trail with CloudTrail, I noticed that the website had been hacked and that part of the hack involved an action during which someone modified the security group settings.

In Task 3, I used a variety of methods to analyze the CloudTrail logs, including the Linux grep utility and the AWS Command Line Interface (AWS CLI).

In Task 4, I used Amazon Athena to search the CloudTrail logs.

In the Challenge section that concluded Task 4, I worked to identify the hacker.

In Task 5, once I discovered the culprit, I removed that user's access. I also took steps to reduce the chances that the AWS account and the Café website would be hacked again.

Business Case Relevance

Martha and Frank were concerned because the website was hacked. They relied on me to discover who did it and to make sure that it would not happen again.

Faythe, Frank, Martha, and others make frequent changes to the website, and sometimes those changes cause issues. Also, that morning, it looked like the website was hacked. Martha and Frank asked me (playing the role of Sofîa) if there was a way to track what was changed and who made the changes.

I became a detective to discover the culprit.

Task 1: Modifying a Security Group and Observing the Website

I started by opening the EC2 service from the Services menu, choosing Compute then EC2.

I selected Instances, and then located and selected the Café Web Server (WebSecurityGroup) instance.

In the Security tab, I chose the sg-xxxxxxxxxx security group.

In the Inbound rules tab, I noticed that only one inbound rule had been defined, which was for HTTP access over TCP port 80.

I chose Edit inbound rules, and then chose Add rule and configured it as follows:

Type: SSH
Port Range: 22
Source: My IP

I made sure to confirm that the TCP port 22 access would be open to only my IP address. The entry showed a CIDR block with my particular IP address followed by /32, not to all IP addresses (which would be 0.0.0.0/0).

Then I chose Save rules.

To observe the Café website, I:

Selected the Café Web Server instance, clicked the Details tab, and copied the Public IPv4 address
Opened a new browser tab and navigated to http://<WebServerIP>/cafe/
Noticed that the website looked normal, with photos appropriate for a bakery café

Task 2: Creating a CloudTrail Log and Observing the Hacked Website

Task 2.1: Create a CloudTrail Log

I went to the AWS Management Console, from the Services menu, selected Management & Governance then CloudTrail. I ignored the AWSOrganizations access denied message at the top of the console.

On the navigation pane on the left, I chose Trails.

I chose Create trail and configured it as follows:

Trail name: monitor (I made sure to use this exact name as specified)
Selected Create a new S3 bucket
For Trail log bucket and folder, I entered monitoring#### (with random digits)
For AWS KMS alias, I entered my initials followed by -KMS

I chose Next, then on the Choose log events page, chose Next again.

On the Review and create page, I chose Create trail.

I verified that my trail appeared on the Trails page.

Task 2.2: Observe the Hacked Website

I returned to the browser tab with the Café website open and refreshed the page.

I had to wait a full minute before the hack occurred. I also pressed and held Shift while choosing the browser refresh button to see the latest changes.

I noticed that the website had been hacked! The image certainly did not look correct for a café.

It was good that I enabled CloudTrail before this happened. CloudTrail could give me valuable information about what users had been doing in the account.

In the AWS Management Console, I browsed to the EC2 service, and observed the Café Web Server instance details.

In the Security tab, I chose the sg-xxxxxxxxxx security group again, and then checked the Inbound rules tab.

I discovered an extra entry! I still saw the entry I created earlier (the rule opening port 22 to only my IP address). But now there was also an additional inbound rule that allowed SSH access from anywhere (0.0.0.0/0).

I knew I needed to search the CloudTrail logs to find out who added this security hole.

Task 3: Analyzing the CloudTrail Logs by Using grep

Task 3.1: Connect to the Café Web Server Host EC2 Instance by Using SSH

I connected to the Café Web Server EC2 instance using SSH.

Task 3.3: Download and Extract the CloudTrail Logs

With my terminal connected via SSH to the Café Web Server EC2 instance, I ran the following command to create a local directory on the web server to download the CloudTrail log files to:

mkdir ctraillogs

I changed to the new directory:

cd ctraillogs

I listed the buckets to recall the bucket name:

aws s3 ls

Then I downloaded the CloudTrail logs (replacing <monitoring####> with the actual bucket name):

aws s3 cp s3://<monitoring####>/ . --recursive

I got a few log files downloaded successfully. I had to wait a bit for logs to populate since CloudWatch posts logs to Amazon S3 every 5 minutes.

I used cd and ls commands to change to the subdirectory where the logs were downloaded (in an AWSLogs/<account-num>/CloudTrail/<Region>/<yyyy>/<mm>/<dd> subdirectory).

I noticed the log files ended in json.gz, indicating they were compressed as GNU zip files.

I extracted the logs:

gunzip *.gz

I ran ls again and saw that all files were now extracted.

Task 3.4: Analyze the Logs by Using grep

To analyze the structure of the logs, I:

Copied one of the file names returned by the ls command
Entered cat in the terminal window, followed by a space, and then pasted the copied file name
Noticed the files were in JSON format but difficult to read

I ran the cat command again, but this time formatted the output:

cat <filename.json> | python -m json.tool

This format was much more readable. I could see the structure of the log entries with standard fields like awsRegion, eventName, eventSource, eventTime, requestParameters, sourceIPAddress, userIdentity, and more.

The number of entries was large, even in just one log file. I needed to find a way to search log entries across multiple files and filter the results.

I started by filtering where the sourceIpAddress matched the IP address of the Café Web Server instance:

ip=<WebServerIP>

Then I ran:

for i in $(ls); do echo $i && cat $i | python -m json.tool | grep sourceIPAddress ; done

This showed several log entries where the sourceIPAddress was the Café Web Server instance.

Next, I ran a similar command to return the eventName of every captured event:

for i in $(ls); do echo $i && cat $i | python -m json.tool | grep eventName ; done

The results contained many describe and list actions that looked relatively harmless, but also some update actions. I realized I might benefit from using a different tool to locate these log entries more easily.

Task 3.5: Analyze the Logs by Using AWS CLI CloudTrail Commands

I opened the AWS CLI Reference page for CloudTrail and checked out the lookup-events command.

I ran the example command to filter the trail for console logins:

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=ConsoleLogin

The results indicated there had been no console login events, or that the only user who had logged in was the same user I was logged in as.

I then ran a command to find actions taken on security groups:

aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceType,AttributeValue=AWS::EC2::SecurityGroup --output text

There were too many results to easily identify the issue, so I narrowed the search to focus on the security group used by the web server instance.

I found the security group ID used by the Café Web Server instance:

region=$(curl http://169.254.169.254/latest/dynamic/instance-identity/document|grep region | cut -d '"' -f4) sgId=$(aws ec2 describe-instances --filters "Name=tag:Name,Values='Cafe Web Server'" --query 'Reservations[*].Instances[*].SecurityGroups[*].[GroupId]' --region $region --output text) echo $sgId

Then I filtered the CloudTrail results using this security group ID:

aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceType,AttributeValue=AWS::EC2::SecurityGroup --region $region --output text | grep $sgId

I realized I might need a better tool for reading these logs, so I decided to try Amazon Athena.

Task 4: Analyzing the CloudTrail Logs by Using Athena

Task 4.1: Create the Athena Table

I went to the CloudTrail console, chose Event history from the navigation pane, and clicked Create Athena table.

For Storage location, I chose the monitoring#### S3 bucket where CloudTrail was storing log files.

I analyzed how the Athena CREATE TABLE statement was formed:

It created a database column for each standard name-value pair in the JSON-formatted CloudTrail logs
The LOCATION statement indicated the S3 location where the table data was stored
I was essentially defining a table schema to parse existing JSON-structured data

After analyzing the CREATE TABLE details, I chose Create table. The table was created with a default name that included the S3 bucket name.

Then I opened the Athena service from the Analytics section of the Services menu.

Task 4.2: Analyze Logs Using Athena

In the Athena Query Editor, I could see the cloudtrail_logs_monitoring#### table in the left panel.

I selected + beside the table to reveal the column names and noticed how each standard element in a CloudTrail log had a corresponding column.

I set up a query results location:

Chose Settings followed by Manage
Set Location of query result to s3://monitoring####/results/
Chose Save

I pasted this SQL query into the Query 1 panel and ran it:

SELECT * FROM cloudtrail_logs_monitoring#### LIMIT 5

This returned five rows of data. I focused on the columns useridentity, eventtime, eventsource, eventname, and requestparameters, which contained the most valuable information.

Then I ran a query that selected only those specific columns, with 30 rows:

SELECT useridentity.userName, eventtime, eventsource, eventname, requestparameters FROM cloudtrail_logs_monitoring#### LIMIT 30

Challenge: Identify the Hacker

Initial Assessment

When I first looked at the challenge to identify the hacker who compromised the website, I noticed that I needed to analyze log data to find who created the security hole in the Café Web Server security group. I would need to experiment with various queries to track down the culprit.

The tips provided were particularly helpful:

Look at the log data returned by queries
Try filtering by events related to the Amazon EC2 service
Remove the LIMIT clause from queries to see all results
Pay attention to the "eventsource" column
Focus on security-related actions in the logs
Use the WHERE clause to search for particular "eventsname" values

Step 1: Understanding the Data Structure

I initially examined the query results shown in the interface. The logs contained several columns including username, eventname, eventsource, and eventtime. This gave me a good overview of the AWS CloudTrail logs I was working with.

Looking at the original data, I could see various users interacting with AWS services, but I needed to filter this down to identify suspicious activity related to security groups.

Step 2: Crafting the Initial Query

I decided to create a query that would show me all security-related events. My first attempt was:

SELECT DISTINCT username, eventname, eventsource FROM cloudtrail_logs_monitoring WHERE eventname LIKE '%Security%' ORDER BY username, eventtime;

However, this query failed with an error: "TABLE_NOT_FOUND: Table 'awsdatacatalog.default.cloudtrail_logs_monitoring' does not exist." This told me I needed to find the correct table name.

Step 3: Finding the Correct Table Name

After some trial and error, I realized the table name needed to be updated. I tried:

SELECT DISTINCT username, eventname, eventsource FROM monitoring4444 WHERE eventname LIKE '%Security%' ORDER BY username, eventtime;

This still returned an error. Looking more closely at the interface, I noticed that a fragment of a working query showed the full table name was "cloudtrail_logs_monitoring4444".

Step 4: Troubleshooting Column Names

With the correct table name, I tried:

SELECT DISTINCT username, eventname, eventsource FROM cloudtrail_logs_monitoring4444 WHERE eventname LIKE '%Security%' ORDER BY username, eventtime;

This gave me a new error: "COLUMN_NOT_FOUND: Column 'username' cannot be resolved."

To troubleshoot this, I ran a simple query to see the structure of the table:

SELECT * FROM cloudtrail_logs_monitoring4444 LIMIT 1;

The results showed that user identity information was stored in a complex JSON-like structure with fields like "assumedRole", "principalId", and "accountId".

Step 5: Running a Successful Query

After understanding the data structure, I modified my query to:

SELECT useridentity.username, eventname, eventsource, eventtime FROM cloudtrail_logs_monitoring4444 WHERE eventname LIKE '%SecurityGroup%' ORDER BY eventtime;

This query successfully returned 51 rows of security group-related events.

Step 6: Analyzing the Results

Looking through the results, I paid close attention to the patterns of activity for different users. I noticed:

Most activities were from "awsstudent" performing actions like:
- DescribeSecurityGroups (querying existing security groups)
- CreateSecurityGroup (creating new security groups)
User "chaos" appeared several times with:
- DescribeSecurityGroups actions (rows 6, 16, 26, 35, 44, 48)
- AuthorizeSecurityGroupIngress (row 47)

Step 7: Identifying the Hacker

The critical finding was in row 47, where user "chaos" executed the AuthorizeSecurityGroupIngress action at 2025-03-05T21:04:50Z. This action modifies security group rules to allow inbound traffic - exactly what would create a security vulnerability.

I recognized this as a classic attack pattern:

Reconnaissance: "chaos" first performed multiple DescribeSecurityGroups actions to understand the environment
Exploitation: Then executed AuthorizeSecurityGroupIngress to create a backdoor by modifying security group permissions

Conclusion

Based on the evidence in the CloudTrail logs, I identified "chaos" as the hacker who created the security hole in the Café Web Server security group. The smoking gun was the AuthorizeSecurityGroupIngress action, which is used to add inbound rules to a security group - creating the vulnerability that allowed unauthorized access.

The methodical approach of examining the log data, focusing on security-related events, and analyzing the sequence of actions helped me successfully identify the culprit behind the hack.

I successfully identified:

The name of the AWS user who created the security hole (chaos)
The exact time they hacked the security group (2025-03-05T21:04:50Z)
The IP address from which they hacked it (I copied this to a text file)
The method they used (programmatic access via AWS CLI)

I had uncovered the identity of the hacker!

Task 5: Analyzing the Hack Further and Improving Security

Task 5.1: Check the OS Users

In my SSH terminal, I ran this command to find out who had recently logged into the OS:

sudo aureport --auth

There was evidence that someone other than ec2-user had logged in - a "chaos-user".

I ran the who command to see who was currently logged in:

who

The user was still logged in! I needed to get them off the instance right away.

I tried to remove the chaos-user OS user:

sudo userdel -r chaos-user

That didn't work because they were still logged in, but it returned the process number they were connected as.

I stopped the process with their active login session:

sudo kill -9 [ProcNum]

I ran who again to verify that chaos-user was no longer connected, then removed the user:

sudo userdel -r chaos-user

I verified no other suspicious OS users who could login:

sudo cat /etc/passwd | grep -v nologin

The root, sync, shutdown, and halt users were all standard OS users in Amazon Linux, so there were no other concerning user logins.

Task 5.2: Update SSH Security

I analyzed SSH settings on the instance:

sudo ls -l /etc/ssh/sshd_config

I noticed the file was modified today! I edited the SSH configuration file:

sudo vi /etc/ssh/sshd_config

I saw on line 61 that password authentication was enabled - definitely not a security best practice! I:

Commented out the "PasswordAuthentication yes" line
Uncommented the "#PasswordAuthentication no" line
Saved the changes and exited the VI editor

I restarted the SSH service:

sudo service sshd restart

In the EC2 console, I updated the Web Server security group settings:

Selected the Web Server security group
Went to the Inbound tab and chose Edit
Deleted the inbound rule allowing port 22 access from 0.0.0.0/0
Saved the change

I had successfully kicked the hacker out of the instance, removed their login account, and updated the SSH settings to allow only users with the correct key pair and my source IP address to connect.